Air gestures refer to device control features performed by a user making movements through the air. One challenge with implementing air gestures is establishing a proper reference frame relative to the user. While a device such as a phone or wearable may be able to determine the location of magnetic North and the direction of gravity, it is more difficult to determine what direction the device user is facing to properly interpret the air gesture. For this reason, current gesture detection implementations may require the device to be in a particular orientation or for a calibration to be performed before each session to accurately detect an air gesture.